Skip to content

Add benchmark matrix with realistic fixtures#7

Closed
intech wants to merge 3 commits intofeat/prototype-estimator-map-oneoffrom
feat/benchmark-matrix
Closed

Add benchmark matrix with realistic fixtures#7
intech wants to merge 3 commits intofeat/prototype-estimator-map-oneoffrom
feat/benchmark-matrix

Conversation

@intech
Copy link
Copy Markdown

@intech intech commented Apr 19, 2026

Summary

Expands benchmark fixture matrix with 6 new realistic schemas. Enables cross-workload performance measurement per tech-lead execution plan Phase 2.

New fixtures (6)

  • otel-metrics.proto — Gauge / Sum / Histogram with DataPoints + AnyValue attributes
  • otel-logs.proto — LogRecord batch export (100 records)
  • k8s-pod.proto — Pod + PodList subset (20 pods, labels, containers, env vars, resource limits)
  • graphql.proto — Request/Response/Error with extensions map
  • rpc-simple.proto — baseline key-value RPC envelope
  • stress.proto — synthetic edge cases (deep nesting, all scalar types, large blobs)

Smoke measurements

Fixture Encoded size toBinary ops/s fromBinary ops/s
SimpleMessage 19 B 187,000 602,000
ExportTraceRequest (100 spans) 35,283 B 136 898
ExportMetricsRequest (50 series) 17,696 B 192 1,211
ExportLogsRequest (100 records) 21,319 B 204 829
K8sPodList (20 pods) 28,900 B 227 1,275
GraphQLRequest 624 B 40,406 208,768
GraphQLResponse 1,366 B 62,282 247,402
RpcRequest 501 B 43,285 236,128
RpcResponse 602 B 65,372 686,342
StressMessage (depth=8, width=200) 12,868 B 1,132 5,049

Base

Stacked on feat/prototype-estimator-map-oneof (PR #4) because fixtures reference AnyValue schema introduced there.

Scope

Internal PR within the Connectum-Framework fork. Pure additive — existing benchmarks unchanged.

🤖 Generated with Claude Code

@intech intech self-assigned this Apr 19, 2026
@intech intech changed the title feat(benchmarks): expand matrix with K8s, GraphQL, RPC, OTel metrics/logs, stress fixtures Add benchmark matrix with realistic fixtures Apr 19, 2026
intech and others added 2 commits April 20, 2026 01:34
…logs, stress fixtures

Prototype series measured only the OTLP traces fixture plus a 3-field
SimpleMessage. Phase 2 of the investigation needs performance readings
across the payload classes protobuf-es actually sees in production, so
that a regression (or improvement) can be attributed to a class of
workload rather than lumped into a single OTLP traces number.

New .proto fixtures under benchmarks/proto/:

- otel-metrics.proto — OTLP metrics export: Gauge/Sum/Histogram oneof,
  NumberDataPoint, HistogramDataPoint with explicit bucket bounds.
- otel-logs.proto — OTLP logs export: LogRecord batch with severity,
  string body, trace/span correlation IDs.
- k8s-pod.proto — Kubernetes Pod list subset: ObjectMeta with
  labels/annotations maps, containers with env/ports/resources,
  container statuses. Map-dominant config payload.
- graphql.proto — GraphQL request/response envelope: long query string,
  map<string,bytes> variables, JSON-in-bytes data + structured errors
  with paths.
- rpc-simple.proto — baseline RPC envelope: routing fields, small
  headers map, opaque bytes payload. Lower bound on per-call overhead.
- stress.proto — synthetic payload: depth-8 self-nesting + 200-wide
  scalar/string/message arrays + 4 KB blob + every proto3 scalar type
  exactly once. Surfaces type-specific regressions that OTLP-only
  fixtures don't hit.

New runner in benchmarks/src/bench-matrix.ts:

- Runs toBinary + fromBinary across all 10 fixtures (including the
  existing SimpleMessage and ExportTraceRequest).
- Emits the standard tinybench table plus a machine-readable JSON
  summary on stdout so CI can diff runs without scraping ANSI output.

Extends benchmarks/src/fixtures.ts with realistic builders for each
shape. Scale knobs (METRICS_SERIES_COUNT, LOGS_RECORD_COUNT,
K8S_POD_COUNT, STRESS_DEPTH, STRESS_ARRAY_WIDTH) are exported so a
follow-up pass can parameterize across multiple sizes without editing
the fixture code.

No changes to existing bench-*.ts files or to the protobuf-es runtime.
The matrix runner is opt-in via `npm run bench:matrix` and the existing
`npm run bench` aggregate is unchanged.

Smoke run confirms every fixture encodes and decodes successfully on
Node v25.8.1 / linux/x64. Encoded sizes span 19 B (SimpleMessage) to
35 KB (OTLP traces).
Adds report.ts entry + helpers for generating comparative benchmark
reports. Inspired by packages/bundle-size/report.ts pattern.

- benchmarks/src/report.ts - runs matrix, generates outputs
- benchmarks/src/report-helpers.ts - SVG chart + markdown table + README injector
- benchmarks/chart.svg - generated comparative chart
- benchmarks/README.md - injected table section with markers

Run via: npm run bench:report

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@intech intech force-pushed the feat/benchmark-matrix branch from 0852010 to 63b86ef Compare April 19, 2026 21:35
- Apply biome format fixes to benchmarks and protobuf files
- Prepend `mkdir -p src/gen-protobufjs` to generate:protobufjs so the
  output directory exists on fresh CI checkouts (pbjs does not create
  parent directories and failed with ENOENT)

Unblocks format, license-header, lint, and attw jobs which all depend
on @bufbuild/protobuf-benchmarks#generate succeeding.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@intech
Copy link
Copy Markdown
Author

intech commented Apr 19, 2026

Contents merged via PR #6 (head branch was re-pointed to feat/benchmark-matrix before merge).

@intech intech closed this Apr 19, 2026
@intech intech deleted the feat/benchmark-matrix branch April 21, 2026 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant